Description of the issue

Currently the CloudWatch Agent does not support IPv6 environments. This means customer using IPv6 only environments will not be able to use the CWA to get telemetry.

Description of changes

This pr allows customers to be able to run the CloudWatch Agent in an IPv6 only environment. Customer simply needs to add use_dualstack_endpoint=true in the agent configuration and what this will do is it will have all the agent endpoint calls use aws dualstack endpoints. A dual stack endpoint will work in either IPv4 or IPv6 so enabling dual stack will allow all aws sdk calls to have dual stack endpoints.

How are we enabling dualstack for all sdk calls?

If the custom sets the use_dualstack_endpoint=true in the agent section, the cloudwatch agent will set the AWS_USE_DUALSTACK_ENDPOINT to true this environment variable will automatically have all sdk used dual stack endpoints if set (as pointed out in this aws doc). This is being done in the toEnvConfig.go file where we see if use_dualstack_endpoint was specified to true and if it was, we'll set the env variable while agent is starting up (following current behavior of setting env variables at agent start up)

There are some extra changes needed to support the CloudWatch Agent fully in IPv6 and those are 1. AMP custom endpoint changes and config-downloader changes.

For the AMP custom endpoint changes, we added a new field in Global_Config of the agent which is currently being used in amp translator. We parse the use_dualstack_endpoint in the apply rule and then set the Global_Config so that the agent knows that the customer set dual stack to true in amp translation. In the amp translator we check Global_Config.UseDualStackEndpoint if set to true, we'll update the amp endpoint to be dualstack endpoint.

IPv4 vs dualstack amp endpoint

IPv4 AMP endpoint

https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/api/v1/remote_write

Dualstack AMP endpoint

https://aps-workspaces.us-east-1.api.aws/workspaces/ws-12345678-1234-1234-1234-123456789012/api/v1/remote_write

the only difference between the endpoint is amazonaws.com is changed to api.aws. Which is exactly what we do in the translator to make amp work in an IPv6 only environment

How about the config downloader IPv6 support?

For the config downloader the agent configuration isn't available as customer uses this to fetch their configuration from ssm parameter store. In this case customer has two options 1. Set the env variable AWS_USE_DUALSTACK_ENDPOINT to true and run the config downloader. Or they can pass in -d option flag to their fetch-config/append-config call and the agent will set the env variable for them. This matches the current pattern used in the agent of using short options like for example -s, -h, etc.

How about cloudwatch agent use_dualstack_endpoint vs env var precedence

CloudWatch Agent Dual-Stack Behavior Matrix (IPv6-only Environment)

Tests

All testing was done on an IPv6 only environment (IPv6 addresses only no IPv4 address) verifying that we are able to get agent metrics and also verified amp and config-dowloader:

AMP testing:

Agent json used

Agent json used for test:

{
  "agent": {
    "metrics_collection_interval": 15,
    "run_as_user": "root",
    "debug": true,
    "region": "us-west-2",
    "use_dualstack_endpoint": true
  },
  "metrics": {
    "metrics_destinations": {
      "amp": {
        "workspace_id": "ws-xxx"
      }
    },
    "metrics_collected": {
      "cpu": {
        "measurement": [
          {"name": "time_active", "rename":"cpu_time_active_renamed"},
          "time_guest", "time_guest_nice", "time_idle", "time_iowait", "time_irq",
          "time_nice", "time_softirq", "time_steal", "time_system", "time_user",
          "usage_active", "usage_guest", "usage_guest_nice", "usage_idle", "usage_iowait", "usage_irq",
          "usage_nice", "usage_softirq", "usage_steal", "usage_system", "usage_user"
        ],
        "metrics_collection_interval": 10
      }
    }
  }
}

Successfully querying amp metrics

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "cpu_usage_system",
          "aws_StorageResolution": "true",
          "cpu": "cpu-total",
          "host": "i-xxx"
        },
        "value": [
          1757692351,
          "0"
        ]
      }
    ]
  }
}

Config downloader testing:

CLI Command used:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -d -a fetch-config -m ec2 -c ssm:/config/standard/linux/ec2 -s

CWA json in ssm parameter store

{
  "agent": {
    "metrics_collection_interval": 15,
    "run_as_user": "root",
    "debug": true,
    "region": "us-west-2",
    "use_dualstack_endpoint": true
  },
  "metrics": {
    "namespace": "testingIPv6-9/12-config-downloader",
    "metrics_collected": {
      "cpu": {
        "measurement": [
          {"name": "time_active", "rename":"cpu_time_active_renamed"},
          "time_guest", "time_guest_nice", "time_idle", "time_iowait", "time_irq",
          "time_nice", "time_softirq", "time_steal", "time_system", "time_user",
          "usage_active", "usage_guest", "usage_guest_nice", "usage_idle", "usage_iowait", "usage_irq",
          "usage_nice", "usage_softirq", "usage_steal", "usage_system", "usage_user"
        ],
        "metrics_collection_interval": 10
      }
    }
  }
}

Successfully getting metrics given in the namespace of the agent json stored in the ssm parameter store

Tested config downloader dual stack changes on Windows

Agent json used:

{
    "agent": {
      "debug": true,
      "region": "us-west-2",
      "use_dualstack_endpoint": true

    },
    "metrics": {
      "namespace": "Windows-config-downloader-test",
      "metrics_collected": {
        "Network Interface": {
          "resources": [
            "*"
          ],
          "measurement": [
            {
              "name": "Bytes Sent/sec",
              "rename": "Bytes_Sent_Per_Sec"
            },
            {
              "name": "Packets Sent/sec",
              "rename": "Packets_Sent_Per_Sec"
            }
          ],
          "metrics_collection_interval": 1
        },
        "procstat": [
          {
            "exe": "amazon-cloudwatch-agent.exe",
            "measurement": [
              "cpu_usage",
              "memory_rss",
              "memory_vms"
            ],
            "metrics_collection_interval": 1
          }
        ]
      },
      "append_dimensions": {
        "InstanceId": "${aws:InstanceId}"
      },
      "force_flush_interval": 10
    },
	"logs": {
		"logs_collected": {
		  "files": {
			"collect_list": [
			  {
				"file_path": "/tmp/test1.log",
				"log_group_name": "{instance_id}",
				"log_stream_name": "{instance_id}/tmp1",
				"timezone": "UTC"
			  }
			]
		  }
		},
        "force_flush_interval": 60
	}
  }

Successful windows Testing outputs

Calling control script with -d dual stack option

use_dual_stack_endpoint set in toml

Getting metrics

Tested Mac config downloader

Tested by checking that mac control script properly config downloader with --dualstack flag.